feat(models): add LoRA support for Qwen-Image-2.1 by Pfannkuchensack · Pull Request #9793 · invoke-ai/InvokeAI

Pfannkuchensack · 2026-10-10T20:49:30Z

Summary

Adds LoRA support to Qwen-Image-2.1.

Stacked on #9787. This PR's own change is the last commit; the one before it is #9787 (#9784 is merged).

Layouts. The published LoRAs target the diffusers module names under a diffusion_model. (ComfyUI), transformer. / base_model.model. (diffusers/PEFT) or lora_unet_ (Kohya) prefix. One exception: ComfyUI fuses each block's gated MLP input into img_mlp.gate_up. It is split by its output rows into gate_layer (first half) and proj (second half), as the base checkpoint converter does. The split covers low-rank layers, PEFT/ai-toolkit DoRA (output-dim magnitude) and full (diff) layers. Full diffs on the RMSNorm weights (attn.norm_q/norm_k, txt_in.text_norm), which ComfyUI's official Turbo LoRA carries, are merged into those weights on every format, since no sidecar can patch a norm.

All or nothing. Identification and conversion share one key table. A LoRA with a layer this transformer lacks, or with a gate_up layer that cannot be split, is not installed as a Qwen-Image-2.1 LoRA. That covers LoKR, LoHA and LyCORIS DoRA, which mix both halves. Applying only the part that fits would pass a partial LoRA off as a whole one.

Backend

New LoRA_LyCORIS_QwenImage21_Config: the gated MLP's names (gate_up, gate_layer, proj, out) or the 4096 width of the block attention (to_q/k/v, to_out.0, low-rank or LoKR), excluding FLUX.2 LoRAs and anything with unsupported keys. Qwen-Image LoRA configs no longer claim such files, attention-only 2.1 LoKr included.
FLUX's Kohya detector now accepts only FLUX's own block names (double_blocks, single_blocks); a Kohya Qwen LoRA whose every key is on the MLP no longer reads as FLUX.
New qwen_image_2_1_lora_loader and qwen_image_2_1_lora_collection_loader nodes. They refuse non-LoRAs and LoRAs of other bases by name, with a hint that Qwen Image LoRAs do not fit 2.1, and skip LoRAs already applied.
qwen_image_2_1_denoise patches the transformer for the whole sampling run, before the prefix KV cache is built:
- plain weights are patched in place and restored afterwards;
- GGUF/int8 weights get sidecars, and fp8 ones per layer;
- working memory is reserved only for sidecars: the LoRA sizes plus one MLP-wide update per forward.

webv2: qwen-image-2-1 joins the LoRA-supported bases. The graph routes the transformer through the collection loader.

Docs: a LoRA section on the Qwen-Image-2.1 page, plus the generate-panel and model-families rows.

Related Issues / Discussions

Part of #9594. Depends on #9787.

QA Instructions

Python: the affected suites pass locally, ruff clean; the full suite runs in CI.
webv2: pnpm lint passes; the generation suites pass.
Docs: the build passes, internal links are valid.
Covered on CPU:
- comfy and Kohya layouts across every block and top-level module, against a meta-device transformer;
- the exact gate_up split, including DoRA magnitudes and full layers;
- all prefixes, and alpha;
- refusals: LoKR/LyCORIS DoRA on gate_up, foreign layers;
- identification against every other LoRA config, the FLUX.2 Klein 9B cases, and a stripped real LoRA fixture;
- the denoise path with a real Linear, merged and as a sidecar: the update is applied from the first step and the weight is unchanged afterwards;
- collection dedupe and refusal messages; the webv2 graph.
GPU (RTX 4090):
- ComfyUI's Turbo LoRA (avg. rank 178, with norm diffs) on the Base model at 8 steps turns the unfinished image into a sharp one on bf16 (merged), GGUF and int8 (sidecars); the sidecar runs land within bf16 rounding of the merged one.
- A Manga-to-Photo LoRA changes GGUF txt2img and Diffusers edits; the Diffusers weights are bit-identical before and after a LoRA run.
- Three community 2.1 LoRAs and the Turbo LoRA install as qwen-image-2-1.

Review

All material findings were resolved, with regression tests:

LoRAs with foreign or unsplittable layers were applied partially; they are now refused at install time and at conversion;
the gate_up split now splits every row-indexed value (low-rank up, DoRA magnitude, full diff), not just the low-rank pair;
working memory was reserved for direct patching too; it is now sized only for sidecars;
the collection loader could apply a LoRA twice;
from GPU testing: the official Turbo LoRA installed as unknown (norm diffs);
from the PR1 review: 2.1 LoRAs on only to_out.0 or the MLP halves, and attention-only LoKr, were filed as Qwen-Image LoRAs.

Remaining limitation: a LoRA on only to_q/to_k/to_v of FLUX.2 Klein 9B's first blocks has exactly the keys and shapes of a Qwen-Image-2.1 one and is identified as the latter. This is pinned in a test and noted in the docs.

Compatibility / Rollout

Generated API files: openapi.json and schema.ts are regenerated; the changes are additive: two new nodes and a new LoRA config. qwen_image_2_1_denoise moves to 1.2.0 without new fields.
Model identification: the new config only claims LoRAs that no other config claimed before.

Checklist

The PR has a short but descriptive title, suitable for a changelog
Meaningful regression coverage added / updated where needed; obsolete tests/code removed
Persisted-state and API changes include required migrations / compatibility validation
Relevant performance/efficiency opportunities considered; material claims have evidence
Material review findings resolved and relevant checks rerun
Documentation added / updated (if applicable)
Updated What's New copy (if doing a release after this PR)

) ## Summary Adds Qwen-Image-2.1 (Base and the 8-step Turbo) as a new `qwen-image-2-1` base. This covers text-to-image plus canvas img2img, inpaint and outpaint. Before this, the released files were misidentified: the GGUF as FLUX schnell, the VAE as SD1. **Model support** - **Formats:** Diffusers folders (`QwenImage21Pipeline`); single-file transformers in bf16, ComfyUI scaled fp8 or `int8_convrot`; GGUF transformers; standalone VAEs in Diffusers or ComfyUI layout. - **Nodes:** model loader, Qwen3-VL 8B text encoder, denoise, i2l and l2i. - **Parity:** the denoise and the decode reproduce `QwenImage21Pipeline` bit for bit. That covers the prompt template and pre-norm hidden states, the Base shift and Turbo sigma table, CPU noise, the scheduler's bf16 step, the prefix KV cache, and the VAE's denormalization and pixel mapping, both done in the VAE's dtype as the pipeline does. - **Strength:** on Base, img2img and inpaint pick the start step by strength (step index), as Qwen-Image, Z-Image and diffusers' img2img pipelines do, so a strength changes the image about as much as on those models (0.75 starts at sigma ~0.85 at 40 steps). Turbo picks by noise level (`clip_timestep_schedule_fractional`): its table is front-loaded, so by index any strength from 0.5 to 0.85 would start above sigma 0.89. - **RGBA VAE:** a decode is saved as RGB when at most 0.1% of pixels have alpha < 224, otherwise as RGBA. Encodes treat the input as opaque. The VAE, bundled or standalone, runs at the configured precision, except float16, which is unstable for it. - **Memory:** encodes and decodes tile up front when the untiled frame would not fit (1024/768/512 px tiles, via the shared Qwen-Image tiling helper at 16x). The working-memory estimates cover the KV cache, CFG, SDPA scores and the GGUF dequant transient, and scale with the VAE and compute dtypes. - **Refused at install** (`InvalidMatchError`): nvfp4 files, torchao-quantized files (such as unsloth's FP8 build; loading them is #9798), int8 files without the convrot marker, and Q8_CR GGUFs. The check is shared with Ideogram 4. - **Identification:** Qwen-Image LoRA configs no longer claim 2.1 LoRAs; attention-only ones are told apart by their 4096 width. The FLUX GGUF config now requires `double_blocks`, so the 2.1 GGUF is no longer taken for FLUX schnell. **Starter models** - Diffusers Base and Turbo, the Q4_K_M GGUF and the ComfyUI VAE. - The Qwen3-VL 8B encoder starter moves to `common.py`, shared with Ideogram 4. `previous_names` keeps existing installs matched. **Frontend** - webv2: base identity, generation policy (VAE and Qwen3-VL 8B encoder are required for non-Diffusers models), graph builder, canvas mapping and metadata modes. - webv1 enums and labels for shared-schema compatibility. **Docs:** a new model page, plus the requirements, model-families, generate-panel, parameters, low-VRAM and FP8 pages. Reference images (Edit) and LoRA support follow in separate PRs. ## Related Issues / Discussions Part of #9594. Builds on #9774 (diffusers 0.41.0). ## QA Instructions - **Python:** the affected suites pass locally, ruff clean; the full suite runs in CI. - **webv2:** `pnpm lint` passes. vitest: 10073 passed; 3 failures are locale-dependent tests that fail on a German-locale machine, unrelated to this change. - **webv1:** `tsc` shows the same 16 errors as on the base commit `9d30e6a148`; none are new. - **Docs:** the build passes and all internal links are valid. - **Parity (RTX 4090):** final latents are bit-identical to `QwenImage21Pipeline` at 1024² and 2048² (40 steps). The decoded 1024² image is RGB bit-identical to the pipeline's (it was off by up to 12/255 on a quarter of the pixels before the dtype fix); a tiled decode differs from the pipeline's untiled one by the tiling. - **Browser (webv2, GGUF Q4_K_M + fp8 encoder + ComfyUI VAE):** - txt2img, about 47 s for 40 steps at 1024². - CFG 4 with a negative prompt. - Recall All, which restores model, VAE, encoder, CFG, seed and prompts. - Canvas img2img, inpaint and outpaint. - A 2048² img2img, where the encode tiles. - **Real files (RTX 4090, 1024², 40 steps, same seed, against the Diffusers transformer):** ComfyUI bf16 is bit-identical; `int8_convrot` 37 dB; GGUF Q4_K_M 21 dB (visibly the same image). The ComfyUI Turbo bf16 file is identified as Turbo and gives clean 8-step images. unsloth's FP8 (torchao) file is refused at install. No ComfyUI scaled-fp8 2.1 build is published; that path is covered by fixtures. ## Review All material findings were resolved, with regression tests: - the Turbo strength mapping; - NaN sigmas at one Base step; - missing encode auto-tiling; - the dtype-blind VAE estimate; - install-time refusal of nvfp4 and unmarked int8; - attention-only LoRA misidentification; - the webv1 enum. From the maintainer review: the decode now matches the pipeline's dtype order (bit-identical image), the bundled VAE honours the configured precision and never runs in float16, the denoise estimate scales with the compute dtype, the Turbo fixture carries the real Turbo scheduler config, and the webv2 builder is tested at CFG > 1 and for the GGUF component requirements. From the second maintainer review: Base image-to-image starts by step index (above), Qwen-Image-2.1 is listed on the Model Format Support page, the builder's unreachable component checks are gone, and tests pin the dtype width at the call site, the strength rounding, the guidance slice and the encoder edge's source field. Remaining limitation: LoRAs that target only `to_out` or the MLP halves, and attention-only LoKr, are told apart from Qwen-Image LoRAs only with #9793. ## Compatibility / Rollout - **Models already installed:** Qwen-Image-2.1 files installed before this PR keep their wrong records (FLUX/SD1) until they are removed and reinstalled. There is no migration. - **Starter rename:** the 8B encoder starter is renamed "Qwen3-VL 8B Encoder" and keeps its old name in `previous_names`. - **Generated API files:** `openapi.json` and `schema.ts` are regenerated; the changes are additive. ## Checklist - [x] _The PR has a short but descriptive title, suitable for a changelog_ - [x] _Meaningful regression coverage added / updated where needed; obsolete tests/code removed_ - [x] _Persisted-state and API changes include required migrations / compatibility validation_ - [x] _Relevant performance/efficiency opportunities considered; material claims have evidence_ - [x] _Material review findings resolved and relevant checks rerun_ - [x] _Documentation added / updated (if applicable)_ - [ ] _Updated `What's New` copy (if doing a release after this PR)_

The prompt encoder reads up to four references through the vision tower; their latents fill the image slots. The encoder reserves memory for the references' attention; the denoise checks slot grids before loading. Reference images are offered only on Diffusers models, which keep the vision tower.

Load ComfyUI, diffusers/PEFT and Kohya LoRAs, split ComfyUI's fused gate_up, and merge norm diffs on every format. Identify 2.1 LoRAs by their gated MLP or 4096-wide attention; refuse any that would apply only in part. Apply them to the denoise transformer through a collection loader, as sidecars on GGUF, int8 and fp8 weights.

…-lora # Conflicts: # docs/src/content/docs/Users Guide/03.Models/03.model-format-support.mdx # invokeai/frontend/api/schema.ts

lstein

Reviewed at f5a5ce6 (own change 3250c61adb..56fe808263). No blockers found. Findings below are non-blocking; the first and fourth are the ones I'd most like addressed.

Non-blockers

Alpha-only module installs, then fails at generation. A valid pair plus a lone diffusion_model.transformer_blocks.1.attn.to_q.alpha is identified as LoRA_LyCORIS_QwenImage21_Config; conversion then raises ValueError: Unsupported lora format: dict_keys(['alpha']) on first render. The pairing check in configs/lora.py (_validate_looks_like_lora, ~L1206) doesn't consider modules with no weight keys at all, so this contradicts the "refused at install" intent.
Sidecar working memory undercounts non-low-rank layers. _lora_working_memory (qwen_image_2_1_denoise.py ~L546) reserves patch_bytes + seq_len * 12288 * 2, which matches the LoRALayer path only. Full diff, DoRA, LoKR and LoHA sidecars go through the aggregation branch of autocast_linear_forward_sidecar_patches (custom_linear.py ~L68-82), which materializes the cast original weight plus an out×in residual on every forward (~100 MB bf16 / ~200 MB fp32 per MLP projection). The 1 GiB base probably absorbs it, but the docstring's "one output-sized update per forward" is inaccurate, and the hard-coded * 2 ignores dtype.itemsize.
Over-reservation under partial load. patch_bytes is reserved whenever the format needs sidecars, but sidecars on CPU-resident layers keep their tensors on CPU (layer_patcher.py ~L336), weight-0 LoRAs are skipped, and norm diffs are merged directly. A large full-diff LoRA on GGUF can force the cache to offload most of the transformer for VRAM that is never used. Slower, not incorrect.
Docs: mixed LoKR/LoHA files do install. qwen-image-2-1.mdx (~L82) and the format-support row say LoKR and LoHA install as unknown. That holds only for pure LoKR/LoHA files: e.g. a LoKR on attn.to_q plus a low-rank attn.to_k pair identifies as Qwen-Image-2.1 and converts both layers. Conversely, a LoRA made only of full diff layers is refused by _has_complete_lora_pair although conversion supports it. Either the docs or the identification rule should change.
Untested lines. Each of these can be removed/altered with the suite still green:
- the working_memory += self._lora_working_memory(...) call in denoise (~L380), and the count_fp8_weights branch (the reserve test covers Diffusers and GGUF only);
- not _is_flux2_lora_state_dict(...) in _is_qwen_image21_lora (~L1192): both Klein tests are already rejected by qwen_image_21_unsupported_lora_keys;
- the anchoring of _BLOCK_ATTENTION_PROJECTION (~L375): the LTX-2 audio_to_video_attn case it guards against isn't exercised;
- the img_mlp.proj. marker (the MLP test always pairs it with img_mlp.out); the odd-row ValueError in the gate_up split; the not-found / not-a-LoRA checks in the loader.

Nits

Two spellings of the same module (e.g. diffusion_model.…to_q and transformer.…to_q, or gate_up plus gate_layer in one block) silently overwrite each other in lora_model_from_qwen_image_21_state_dict; a collision check would keep "never a partial LoRA" honest.
A diff plus a low-rank pair on the same module: any_lora_layer_from_state_dict builds the low-rank layer and drops the diff.
The denoise node doesn't check the LoRA's base, so a Qwen-Image (non-2.1) loader wired into 2.1 denoise in the workflow editor applies whatever shapes fit. Same as other families, so not a regression.
The single loader raises on an already-applied LoRA while the collection loader skips it silently. Both avoid double application.
test_another_familys_lora_is_named_without_the_qwen_image_hint asserts the exact error string; test_loras_reserve_room… restates the formula rather than checking behaviour.

Verified clean

Weights restored on exceptions and cancellation (ExitStack ordering, apply_smart_model_patches finally); no in-place mutation of cached_weights / RAM-cache copies; separate storage for the norm pass.
QwenImage21KVCache is built after patching and lives for one run, so LoRA and non-LoRA runs can't share state.
gate_up half order matches split_fused_gate_up and diffusers' converter; PEFT DoRA row split is numerically exact against the fused reference; rank/alpha scaling preserved; LyCORIS DoRA / LoKR / LoHA on gate_up and odd row counts are refused; bias keys are refused rather than dropped.
Kohya names resolve via the lookup table (no underscore ambiguity); foreign keys refuse the file.
The FLUX Kohya regex narrowing still covers real FLUX layouts (including TE-only); no model in tests/model_identification changes identity.
webv2: isLoraCompatibleWithModel requires an exact base match, so Qwen-Image and 2.1 LoRAs don't cross; graph unchanged with no active LoRAs.
openapi.json regenerates identically; the API changes are additive.
Focused pytest (nodes, configs, LoRA conversions, model identification) and the webv2 generation/canvas vitest suites pass; ruff clean.

Not checked: GPU/fp8 runs, VRAM peaks, webv2 tsc/lint, schema.ts regeneration.

Every module must be one complete low-rank, DoRA, LoKR, LoHA or full layer, so pure LoKR/LoHA/diff files install and partial ones stay unknown; Qwen Image no longer claims LTX-2 or 4096-wide LoKR files. Sidecar working memory is priced per layer kind and also covers partly loaded merged formats; DoRA on an int8 transformer is refused before loading.

Pfannkuchensack · 2026-10-22T05:37:51Z

Thanks for the review. All findings and nits are addressed in 381f461, except the two you marked as matching other families. The rule is now: a file is identified as a Qwen-Image-2.1 LoRA exactly when all of it applies. unsupported_keys owns that rule for both identification and conversion.

1. Alpha-only module.
unsupported_keys now checks every module: its keys must form exactly one complete layer (low-rank, DoRA, LoKR, LoHA or full diff). It refuses:

alpha-only and magnitude-only modules, half pairs, and mixed kinds on one module;
two spellings that land on the same value;
a gate_up beside its own block's gate_layer/proj;
block indices outside 0–31 or with leading zeros.

The odd-row check moved there, so _split_gate_up no longer raises.

A test feeds each defect to every LoRA config, in three layouts (ComfyUI gate_up, PEFT attention, attention-only LoKR); no config claims any of them. Such files install as unknown models.

2. Sidecar working memory.
It is now priced per layer kind, in the compute dtype's itemsize:

low-rank adds the output plus rows × rank;
the aggregation path adds weight copies: full 2, LoKR 3, LoHA 4, DoRA 5, and each further layer summed on the same module adds up to 2 more;
Linears that read the timestep embedding are priced at 2 rows, not the sequence length.

Measured on an RTX 4090 and an RX 9060 XT, for bf16, fp8 and GGUF Q4 weights on modulation.1 and gate_layer: the reserve is at or above the measured peak of the update path in every case. It over-reserves by at most about two weight copies (≤ 256 MiB on modulation.1). DoRA is priced one copy above CUDA's measurement because ROCm needs it.

A CPU test checks the reserve against the allocations of a real sidecar forward, for each kind and for stacks of up to four layers.

3. Over-reservation under partial load.
Norm diffs and weight-0 LoRAs are no longer counted. Which layers stay on the CPU is not known before the transformer is locked, so sidecar formats still count all patch tensors.

Merged formats previously reserved nothing. Partial loading sidecar-patches every CPU-resident Linear, so they now reserve one module's worth (its patch plus its update).

4. LoKR/LoHA docs vs identification.
The identification side changed. Pure LoKR, LoHA, DoRA and full-diff files now install when they apply in full. LoHA and diff count as direct width evidence; LoKR's factored width counts unless the file names a module that only Qwen Image has.

Tucker cores (lokr_t2, hada_t1/t2) are refused, because LyCORIS writes them only for convolutions. The docs state the rule. They also say that DoRA does not run on an int8 transformer; that case is now refused with a clear message before the transformer loads.

5. Untested lines.
Each now has a test that fails when the line is removed:

the working-memory call and the fp8 branch;
the LoKR, LoHA and diff width evidence;
the img_mlp.proj. marker on its own;
the odd-row refusal;
the loader's not-found and not-a-LoRA checks.

Two of your items turned out redundant, and are handled as follows:

_is_flux2_lora_state_dict is removed from the 2.1 check, because every key it decides on is already refused by unsupported_keys.
The anchoring of _BLOCK_ATTENTION_PROJECTION stays, but it has no observable effect, so it has no test of its own.

While checking the anchoring I found something the PR had introduced: the Qwen-Image (v1) config claimed LTX-2 LoRAs with a "" or transformer. prefix, so the result depended on probe order. v1 now refuses has_ltx2_lora_keys explicitly. The probe-independence test covers four prefixes.

Nits

Colliding spellings, and a diff next to a pair on one module: refused (see 1).
Base check in the denoise, and the single loader raising on an already-applied LoRA: unchanged, as in the other families.
The hint test now asserts the meaningful parts of the message. The reserve test now checks behaviour: the exact reserve for known layers, merged versus sidecar formats, and sum versus max across modules.

The FLUX.2 Klein 9B overlap is now documented precisely:

An attention-only Klein LoRA still reads as a 2.1 one.
One that also targets proj_out or norm_out.linear installs too, but those layers do not fit, because the widths differ.
A file on only those two layers is not recognised as 2.1.

Checks: focused pytest (configs, LoRA conversions, model identification, 2.1 nodes, autocast) passes, and ruff is clean. The webv2 graph is unchanged.

Pfannkuchensack force-pushed the feat/qwen-image-2-1-lora branch from 57cb054 to 2184ae7 Compare October 10, 2026 23:56

This was referenced Oct 10, 2026

feat(models): add Qwen-Image-2.1 text-to-image and canvas support #9784

Merged

[enhancement]: Load torchao-quantized single-file transformers (unsloth FP8) #9798

Open

Pfannkuchensack marked this pull request as draft October 11, 2026 00:20

Pfannkuchensack force-pushed the feat/qwen-image-2-1-lora branch from 2184ae7 to e23bbbc Compare October 11, 2026 00:27

Pfannkuchensack marked this pull request as ready for review October 11, 2026 01:01

Pfannkuchensack added 2 commits October 19, 2026 06:39

Pfannkuchensack force-pushed the feat/qwen-image-2-1-lora branch from e04ef43 to 56fe808 Compare October 19, 2026 04:51

Merge remote-tracking branch 'upstream/main' into feat/qwen-image-2-1…

f5a5ce6

…-lora # Conflicts: # docs/src/content/docs/Users Guide/03.Models/03.model-format-support.mdx # invokeai/frontend/api/schema.ts

Pfannkuchensack assigned joshistoast Oct 21, 2026

Pfannkuchensack requested a review from joshistoast October 21, 2026 19:39

lstein reviewed Oct 22, 2026

View reviewed changes

Merge branch 'main' into feat/qwen-image-2-1-lora

0d20630

Navigation Menu

Repository navigation

feat(models): add LoRA support for Qwen-Image-2.1 - #9793

feat(models): add LoRA support for Qwen-Image-2.1#9793
Pfannkuchensack wants to merge 5 commits into
invoke-ai:mainfrom
Pfannkuchensack:feat/qwen-image-2-1-lora

Pfannkuchensack commented Oct 10, 2026 •

edited

Loading

Uh oh!

lstein left a comment

Uh oh!

Pfannkuchensack commented Oct 22, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Pfannkuchensack commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Issues / Discussions

QA Instructions

Review

Compatibility / Rollout

Checklist

Uh oh!

lstein left a comment

Choose a reason for hiding this comment

Non-blockers

Nits

Verified clean

Uh oh!

Pfannkuchensack commented Oct 22, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Pfannkuchensack commented Oct 10, 2026 •

edited

Loading